Abstract
Introduction: Patients with hematological malignancies face a high risk of infectious complications due to immunosuppressive treatments and disease-related factors. Managing these infections requires prompt, accurate, and evidence-based clinical decisions, often in dynamic and complex scenarios. Recent advances in artificial intelligence (AI), particularly large language models (LLMs), offer promising support tools in this domain. Van de Wyngaert et al. (Haemophilia 2023; doi:10.1111/hae.14858) emphasized that ChatGPT-4o may be valuable for clinicians and patients but must not replace medical judgment. There are examples of use of ChatGPT4o® in hematology (http://codigorojo.tech/home/)( more than 200,000 consultations in the last 6 months). Red Code (Código Rojo) has developed ultra-specialized chatbots in, for example, Myeloma and Lymphoma in collaboration with SEHH (Spanish Society of Hematology and Hemotherapy, HematoBot®, www.sehh.es). Building upon this, our project aims to develop and evaluate an ultra-specialized AI chatbot for infection management in oncohematology.
Objectives: The primary objective is to develop and validate a multilingual, ultra-specialized chatbot—OncoInfeccAPP—based on LLMs (ChatGPT4o, OpenAI®), designed to deliver accurate, context-aware support for managing infections in hematologic malignancies. A secondary objective is to compare its performance with ChatGPT-4o, with a focus on clinical accuracy, completeness, contextualization, and ethical compliance.
Materials and Methods: OncoInfeccAPP was developed using a Retrieval-Augmented Generation (RAG) framework integrating expert-curated resources such as NCCN 2023, ECIL-10, and other guidelines. The chatbot supports real-time document processing, multilingual voice interaction, and image analysis and is adaptable to local clinical protocols and legal frameworks. The chatbot is continuously updated by hematology experts to ensure content validity and relevance. A standardized set of prompts and 120 questions across various difficulty levels was designed by two independent hematologists. These questions address bacterial, fungal, and viral infections in neutropenic and non-neutropenic settings. Responses from both OncoInfeccAPP and ChatGPT-4o are blinded and scored independently by two infectious disease experts using an 8-domain rubric: clinical accuracy, completeness, clarity, relevance, consistency, contextualization, ethics, and safety.
Results: Preliminary internal testing has shown that OncoInfeccAPP can deliver rapid, accurate, and personalized responses aligned with current guidelines. In the initial set of 25 clinical scenarios, OncoInfeccAPP achieved a mean score of 8.4/10 in clinical accuracy, outperforming ChatGPT-4o (7.5/10), particularly in contextualization and ethical compliance. Full results from independent expert evaluation are in progress and will be presented.
Conclusions: OncoInfeccAPP is a novel AI-based clinical decision support tool that addresses the growing need for timely, guideline-concordant infectious disease management in hematology. Its ability to integrate multilingual, personalized, and legally compliant support highlights its utility as both a clinical and educational resource. In addition to aiding clinical care, such tools may enhance institutional standardization, antimicrobial stewardship, and reduce practice variability. However, current limitations include dependency on the quality and scope of the underlying corpus, potential biases inherited from source materials or the LLM itself, and the need for prospective external validation to confirm generalizability and real-world clinical utility.
